Image processing apparatus, image processing method, computer readable medium, and computer program thereof

ABSTRACT

It is an object of the present invention to acquire 3-dimensional image data that correctly express a structural configuration of the subject by processing the projected image data obtained by projecting the subject toward the plural angles . A shape computing unit  260  calculates shape data which represent a outline of the subject, using a plurality of 1st image data of the subject obtained by projecting the subject toward the plural angles. A distributing unit  220  generates gray-scale density-distributions by spreading and distributing densities shown in gray-scale image of the subject respectively within angles along which said gray-scale image was captured. A second integrating unit  240  generates 3-dimensional gray-scale data, which expresses the subject with the 3-dimensional image data, by integrating a plurality of said gray-scale density-distributions generated from a plurality of gray-scale image captured from a plurality of angles. A first integrating unit  280  integrates the shape data and 3-dimensional gray-scale image. A data processing unit  300  generates 3-dimensional image data by extracting only density that exists in perimeter of a shape that the shape data represent, from the 3-dimensional gray-scale data.

[0001] This present application claims priority from a Japanese PatentApplication No. 2000-381636 filed on Dec. 15, 2000, and a JapanesePatent Application No. 2001-120262 filed on Apr. 18, 2001, the contentsof which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

[0002] Structure of a subject along an image-capturing direction, thatis, information along a depth direction, cannot be acquired only by oneprojected image obtained by a transmission electron microscope and thelike. In this case, information along the depth direction of the subjectcan be acquired by spreading gray scale intensities of a plurality ofprojected images captured from a plurality of angles within theimage-captured angles, integrating them together, and examining theintegration thereof.

[0003] However, by a simple integration of gray-scale intensities of aplurality of projected images, which was captured from a plurality ofangles and spread within the image-captured angles, integrations of thegray-scale intensities may arise at the positions where the subjectoriginally did not exist, to give false information, which shows as ifthe subject existed there. Conventionally, it was difficult to suppressor remove the above-described false information. Accordingly, it wasimpossible to acquire 3-dimensional image data that correctly expressthe structural configuration of the subject by processing the projectedimage data obtained by projecting the subject toward the plural angles.

SUMMARY OF THE INVENTION

[0004] Therefore, it is an object of the present invention to provide animage processing apparatus, an image processing method, a computerreadable medium, and a computer program thereof, which are capable ofovercoming the above drawbacks accompanying the conventional art. Thisobject is achieved by combinations described in the independent claims.The dependent claims define further advantageous and exemplarycombinations of the present invention.

[0005] According to the first aspect of the present invention, an imageprocessing apparatus for acquiring 3-dimensional data regarding astructural configuration of a subject by processing image data of thesubject is provided. The image processing apparatus includes a shapecomputing unit which calculates shape data which represent an outline ofthe subject, using a plurality of 1st image data of the subject obtainedby projecting the subject toward the plural angles, a first integratingunit which integrates the shape data and the gray-scale image of thesubject acquired by a projection-type image-capturing apparatus.

[0006] The image processing apparatus may further include a distributingunit which generates gray-scale density-distributions by spreading anddistributing densities shown in the gray-scale image respectively withinangles along which the gray-scale image was captured, and a secondintegrating unit which generates 3-dimensional gray-scale data, whichexpresses the subject with 3-dimensional image data, by integrating aplurality of the gray-scale density-distributions generated from aplurality of the gray-scale image captured from a plurality of angles,wherein the first integrating unit integrates the shape data and the3-dimensional gray-scale data by the second integrating unit. In thiscase, the image processing apparatus may further include a dataprocessing unit which generates 3-dimensional image data by extractingonly density, which exists in perimeter of the shape that the shape datarepresent, from the 3-dimensional gray-scale data.

[0007] The image processing apparatus may further include a gray-scaledensity data-generating unit which calculates the gray-scale image whichshould be acquired when the subject was captured from a different anglefrom the plurality of the above angles, from the above 3-dimensionalimage data.

[0008] The image processing apparatus may further include a samplingunit which extracts respectively gray-scale image of a plurality of thesubjects facing a plurality of angles, included in a single gray-scaleimage acquired by the projection-type image-capturing apparatus, whereinthe first integrating unit may use a plurality of the gray-scale imagewhich the sampling unit extracted.

[0009] The subject may constitutes substance which has a helicalsymmetry, the apparatus dealing with gray-scale image of the substanceas the above single gray-scale image, further including an anglecomputing unit which obtains a reference angle which the single subjectas a reference faces, and further calculates angles which each of thesubjects faces by adding or subtracting the angle by which the helixrotates by the single subject, from the reference angle.

[0010] The second aspect of the present invention provides an imageprocessing apparatus for acquiring 3-dimensional data regarding astructural configuration of a subject by processing image data of thesubject, is provided. The image processing apparatus includes a shapecomputing unit which calculates shape data which represent an outline ofthe subject, using a plurality of 1st image data of the subject obtainedby projecting the subject toward the plural angles, and an image datagenerating unit which generates 3-dimensional image data by distributingdensity of the gray-scale image of the subject acquired by aprojection-type image-capturing apparatus, around the outline.

[0011] The image processing apparatus may further include a gray-scaledensity data-generating unit which calculates the gray-scale image whichshould be acquired when the subject was captured from a different anglefrom the plurality of the above angles, from the above 3-dimensionalimage data.

[0012] The image processing apparatus may further include a samplingunit which extracts respectively gray-scale images of a plurality of thesubjects facing a plurality of angles, included in a single gray-scaleimage acquired by the projection-type image-capturing apparatus, whereinthe image data generating unit may use a plurality of the gray-scaleimages which the sampling unit extracted.

[0013] The subject may constitutes substance which has a helicalsymmetry, while the apparatus deals with data gray-scale image of thesubstance as the single gray-scale image, further including an anglecomputing unit which calculates an angle which the single subject as areference faces, further calculating angles which each of the subjectsfaces by adding or subtracting the angle which the helix rotates by thesingle subject, from the reference angle.

[0014] Moreover, in the 1^(st) and 2^(nd) aspect of the presentinvention, the gray-scale image may be image data acquired by atransmission electron microscope. An image data of the pattern of thesubject may be used as the first image data.

[0015] The shape computing unit may include a position relating unitwhich relates positions of processing objective points in the subject ina 1st and 2nd images which are consecutive in the order of angle, andrelates the positions of the processing objective points in the 2ndimage and a 3rd image, a height data computing unit which calculatesheight data for representing height of the position of the processingobjective point using two sets of position relations by the positionrelating unit, and a shape data computing unit which generates the shapedata of the subject using the height data of a plurality of theprocessing objective points by the height data computing unit, whereinthe apparatus calculates the shape data using three or more of the firstimages. In this case, the 1st, 2nd, and 3rd images may be imagescaptured with a rotation of the subject by a predetermined angle with asame rotation axis as the rotation center, and the height data computingunit may calculate the height data using the predetermined angle.Besides, the 1st and 2nd image may be images captured with a rotation ofthe subject by a predetermined angle with a first rotation axis as therotation center, and the 3rd image may be an image captured with arotation of the subject by the predetermined angle from the positionwhere the 1st image was captured, with another rotation axis as therotation center, and the height data computing unit may calculate theheight data using the predetermined angle. The position relating unitmay calculate distances from a base line defined in advance in thesubject to each of the processing objective points in the image, andre-detects positions of the processing objective points in the 1st and2nd images, when a difference of the distance in the 1st and 2nd imagesis more than a predetermined value, and relates again the processingobjective points in the 2nd and 3rd images when a difference of thedistance in the 2nd and the 3rd images is more than a predeterminedvalue. The height data computing unit may calculate the height data by aleast-squares error method with height data of the 1st and 2nd imagesand height data by the 2nd and 3rd images. Further to this case, theposition relating unit may re-detect the positions of the processingobjective points, when a error by the least-squares error method is morethan a predetermined value.

[0016] The third aspect of the present invention provides an imageprocessing method for acquiring 3-dimensional data of a subject byprocessing image data of the subject. The image processing methodincludes steps of calculating shape data which represent an outline ofthe subject, using a plurality of 1st image data of the subject obtainedby projecting the subject toward the plural angles, generatinggray-scale density-distributions by spreading and distributingdensities, shown in gray-scale image of the subject acquired by aprojection-type image-capturing apparatus, respectively within anglesalong which the gray-scale image was captured, generating 3-dimensionalgray-scale data, which expresses the subject with 3-dimensional imagedata, by integrating a plurality of the gray-scale density-distributionsgenerated from a plurality of the gray-scale image captured from aplurality of angles, integrating the shape data and the 3-dimensionalgray-scale data, and generating 3-dimensional image data by extractingonly density which exists in perimeter of the outline which the shapedata represent, from the 3-dimensional gray-scale data.

[0017] The image processing method may further includes steps ofextracting gray-scale image of a plurality of the subjects facing aplurality of angles from a single gray-scale image acquired by theprojection-type image-capturing apparatus, and generating the gray-scaledensity-distributions using the extracted gray-scale image. In thiscase, the subject may constitute substance which has a helical symmetry,while the apparatus deals with gray-scale image of the substance as thesingle gray-scale image. The image processing method may further includesteps of recognizing a reference angle, which is an angle the singlesubject as a reference faces, and calculating angles which each of thesubjects faces by adding or subtracting the angle which the helixrotates by the single subject from the reference angle.

[0018] The forth aspect of the present invention provides an imageprocessing method for acquiring 3-dimensional data of a subject byprocessing image data of the subject. The image processing methodincludes steps of calculating an outline of the subject, using aplurality of 1st image data of the subject obtained by projecting thesubject toward the plural angles, and generating 3-dimensional imagedata by distributing density of the gray-scale image of the subjectacquired by a projection-type image-capturing apparatus respectively,around the outline.

[0019] The image processing method may further include steps ofextracting gray-scale image of a plurality of the subjects facing aplurality of angles, from a single gray-scale image acquired by theprojection-type image-capturing apparatus, and generating 3-dimensionalimage data by distributing respectively the extracted gray-scale image,around the outline. In this case, the subject may constitute substancewhich has a helical symmetry, while the apparatus deals with gray-scaleimage of the substance as the single gray-scale image. Further to thiscase, the image processing method may further include steps ofrecognizing a reference angle, which is an angle the single subject as areference faces, and calculating angles which each of the subjects facesby adding or subtracting from the reference angle, the angle which thehelix rotates by the single subject.

[0020] The fifth aspect of the present invention provides a computerexecutable program for image processing for acquiring 3-dimensional dataof a subject by processing image data of the subject. The programincludes a shape computing unit operable to calculate shape data whichrepresent a structural configuration of the subject, using a pluralityof 1st image data of the subject obtained by projecting the subjecttoward the plural angles, and a first integrating unit operable tointegrate the shape data and gray-scale image of the subject acquired bya projection-type image-capturing apparatus.

[0021] The sixth aspect of the present invention provides a computerexecutable program for image processing for acquiring 3-dimensional dataof a subject by processing image data of the subject. The programincludes a shape computing unit operable to calculate an outline of thesubject, using a plurality of image data of the subject obtained byprojecting the subject toward the plural angles, and an image datagenerating unit operable to generate 3-dimensional image data bydistributing density of the gray-scale image of the subject acquired bya projection-type image-capturing apparatus, around the outline.

[0022] The seventh aspect of the present invention provides acomputer-readable medium which stores therein a program for imageprocessing for acquiring a 3-dimensional data of a subject by processingan image data of the subject. The program includes a shape computingunit operable to calculate a shape data which represent an structuralconfiguration of the subject, using a plurality of 1st image data of thesubject obtained by projecting the subject toward the plural angles anda first integrating unit operable to integrate the shape data and agray-scale image of the subject, acquired by a projection-typeimage-capturing apparatus.

[0023] The eighth aspect of the present invention provides acomputer-readable medium which stores therein a program for imageprocessing for acquiring a 3-dimensional data of a subject by processingan image data of the subject. The program includes a shape computingunit operable to calculate an outline of the subject, using a pluralityof 1st image data of the subject obtained by projecting the subjecttoward the plural angles, and an image data generating unit operable togenerate 3-dimensional image data by distributing a density of thegray-scale image of the subject, acquired by a projection-typeimage-capturing apparatus, around the outline.

[0024] This summary of invention does not necessarily describe allnecessary features so that the invention may also be a sub-combinationof these described features.

BRIEF DESCRIPTION OF THE DRAWINGS

[0025]FIG. 1 shows a configuration of an image processing apparatus 200.

[0026]FIG. 2 shows a configuration of a shape computing unit 260.

[0027]FIG. 3 shows an example of a hardware block diagram of an imageprocessing apparatus 200.

[0028]FIG. 4 is a figure explaining how to acquire gray-scale image.

[0029]FIG. 5 is a figure explaining how to acquire gray-scale image.

[0030]FIG. 6 is a figure explaining how to acquire gray-scale image.

[0031]FIG. 7 shows longitudinal-section information on an example of3-dimensional gray-scale data.

[0032]FIG. 8 shows another example of the longitudinal-sectioninformation on the 3-dimensional gray-scale data.

[0033]FIG. 9 shows the longitudinal-section information on an example ofa integrated image by a first integrating unit 280.

[0034]FIG. 10 shows the longitudinal section of an example of the3-dimensional image data by a data processing unit 300.

[0035]FIG. 11 shows an example of operation by a gray-scale densitydata-generating unit 340.

[0036]FIG. 12 is a figure for explaining a second example of operationby an image processing apparatus 200.

[0037]FIG. 13 is a figure for explaining a second example of operationby an image processing apparatus 200 FIG. 14 shows a configuration of animage processing apparatus 201 which is a second embodiment of thepresent invention.

[0038]FIG. 15 shows a particle 491, which is an example of the subjectconstituting the substance 490, which has a helical symmetry.

[0039]FIG. 16 shows an example of operation by a sampling unit 210 andan angle computing unit 215.

[0040]FIG. 17 shows other examples of operation by a sampling unit 210and an angle computing unit 215.

DETAILED DESCRIPTION OF THE INVENTION

[0041] The invention will now be described based on preferredembodiments, which do not intend to restrict the scope of the presentinvention, but rather to exemplify the invention. All of the featuresand the combinations thereof described in the embodiments are notnecessarily essential to the invention.

The First Embodiment

[0042]FIG. 1 shows a configuration of an image processing apparatus 200,which is a first embodiment of the present invention. The imageprocessing apparatus 200 includes a distributing unit 220, a secondintegrating unit 240, a shape computing unit 260, a first integratingunit 280, a data processing unit 300, an image data generating unit 320,a gray-scale density data-generating unit 340, and an outputting unit360. The image processing apparatus 200 is an apparatus which acquires3-dimensional information regarding structural configuration of asubject using 1st image data obtained by projecting the subject towardthe plural angles, and a plurality of projected image data, that is,gray-scale image acquired by the transmission electron microscope andthe like. A gray-scale image can also be used as the 1st image data.

[0043] The distributing unit 220 generates gray-scaledensity-distributions by spreading and distributing respectively aplurality of gray-scale image acquired from external, within capturedangles, that is, within projected angles, and outputs them to the secondintegrating unit 240. For example, the distributing unit 220 distributesdensities of gray-scale image equally along the direction of spreading.

[0044] The second integrating unit 240 generates 3-dimensionalgray-scale data by integrating a plurality of gray-scaledensity-distributions by the distributing unit 220, maintaining theangles, and outputs the information to the first integrating unit 280.The generated 3-dimensional gray-scale data corresponds to informationwhich expresses a subject with 3-dimensional image data- Falseinformation may be included in the 3-dimensional gray-scale data here.

[0045] A shape computing unit 260 calculates height information onspecific points of the subject, that is, processing objective points,from three or more image data captured from three or more angles, andcalculates shape data which represent outline of the subject using theheight information on a plurality of the processing objective points.

[0046]FIG. 2 shows an example of a configuration of the shape computingunit 260. The computing unit 260 includes a position relating unit 262,a height data computing unit 264, and a shape data computing unit 266.

[0047] The position relating unit 262 recognizes positions of aplurality of processing objective points in two set of the image datawhich is consecutive in the order of an angle, relates themrespectively, and outputs them to the height data computing unit 264.

[0048] In detail, relating positions of the processing objective pointsin 1st image data and 2nd image data, which is consecutive in order ofangles, and relating positions of the processing objective points in the2nd image data and the 3rd image data just next to the 2nd in order ofangles. This relating process is performed sequentially also to theother image combinations such as the 3rd and 4th image data, and so on.Each of the processing objective points is related in 2 or more sets ofimage combination, respectively.

[0049] Moreover, the position relating unit 262 defines a base line inthe image data, and measures distances of the base line and theprocessing objective points. When a difference of the distances from thebase line to the processing objective points in two images, whichconstitute an image combination, is more than a predetermined level, theposition relating unit performs again the position recognitions and therelations to the processing base line.

[0050] Here, the position relating unit relates the positions ofprocessing objective points using the above mentioned predeterminedangle when each of the images are captured with a rotation of thesubject by the predetermined angle with the same rotation axis as therotation center, or the 1st and 2nd images among three or more imagedata, are captured with a rotation of the subject by a predeterminedangle with a rotating axis as a rotating center, and the 3rd image arecaptured with a rotation of the subject with another rotating axis as arotating center, by the predetermined angle, from the position where the1st image was captured.

[0051] A height data computing unit 264 calculates height information ofthe processing objective points related by the position relating unit262 in two set of image data respectively, and outputs them to the shapedata computing unit 266. The processing performed by the height datacomputing unit 264 is same as the so-called processing of a stereoscopicvision method. Here, since each of the processing objective points isrelated in at least 2 or more sets of image combination respectively,the height data computing unit 264 calculates two or more heightinformation of each of the processing objective points. The height datacomputing unit 264 calculates height information which should beoutputted by the least-squares error method based on a plurality ofheight information. When the error by the least-squares error method ismore than a predetermined value, the position relating unit 262re-detects the positions of processing objective points.

[0052] A shape data computing unit 266 calculates shape data of thesubject using height information of the processing objective pointswhich the height data computing unit 264 calculated, and planeinformation of the processing objective points calculated from the imagedata. The shape data computing unit 266 outputs the calculated shapedata to the image data generating unit 320, the first integrating unit280, and the outputting unit 360.

[0053] Returning to FIG. 1, the first integrating unit 280 generates aintegrated image by integrating shape data by the shape computing unit260 on 3-dimensional gray-scale data by the second integrating unit 240,and outputs the image to the data processing unit 300.

[0054] The data processing unit 300 generates 3-dimensional image databy extracting only information which exists around the shape data orinformation which lap with the shape data, from 3-dimensional gray-scaledensity-distributions, using the integrated image which the firstintegrating unit 280 outputted, and outputs the 3-dimensional image datato the gray-scale density data-generating unit 340 and the outputtingunit 360. When the false information is included in the 3-dimensionalgray-scale data, there is no lapping between the false information andthe shape data. Therefore, false information is not included in theinformation that the data processing unit 300 extracts. Consequently,the 3-dimensional image data correspond to information representing a3-dimensional structure of the subject with more accuracy.

[0055] The image data generating unit 320 generates 3-dimensional imagedata of the subject by distributing the gray-scale image acquired fromthe external, around the corresponding part of the outline which isshown by the shape data which the shape computing unit 260 calculated,and outputs the generated 3-dimensional image data to the gray-scaledensity data-generating unit 340 and the outputting unit 360. Forexample, an image data generating unit 320 generates the 3-dimensionalgray-scale image by distributing densities of the gray-scale imageevenly only over the area which is in a predetermined range from thepart of the outline.

[0056] Here, false information does not arise because the image datagenerating unit 320 generates 3-dimensional image data of the subject,without going through processes of spreading and integrating thegray-scale image, within the captured angle.

[0057] The gray-scale density data-generating unit 340 generates aprojected image captured from an angle specified using the 3-dimensionalimage data which the image data generating unit 320 or the dataprocessing unit 300 generated, that is, gray-scale image of the subject,and outputs the data to the outputting unit 360. A gray-scale densitydata-generating unit 340 acquires the specified angle through externalinput device, such as a keyboard.

[0058] The outputting unit 360 outputs the shape data by the shapecomputing unit 260, the 3-dimensional image data by the image datagenerating unit 320, the 3-dimensional image data by the data processingunit 300, and the gray-scale image by the gray-scale densitydata-generating unit 340 to an external printer or external display andthe like.

[0059]FIG. 3 shows an example of the hardware block diagram of an imageprocessing apparatus 200. In the present embodiment, the imageprocessing apparatus 200 has CPU (central processing unit) 602, ROM(read only memory) 604, RAM (random access memory) 606, display 608, aprinter 610, an input device 612, a hard disk drive unit 614, the FD(floppy disk) drive 616, and the CD-ROM (compact disk ROM) drive 618.

[0060] The CPU 602 performs processing based on the program stored inthe RAM606 and the ROM604. The display 608 displays various information.A printer 610 prints various information. The input device 612 inputssetup and the like for the image processing apparatus 200. The FD drive616 reads data or a program from the floppy (registered trademark) disk620, and passes them to CPU 602. The CD-ROM drive 618 reads data or aprogram from the CD-ROM 622, and passes them to CPU 602. The hard disk614 stores the data or the program read by the FD drive 616 or CD-ROMdrive 618, and the data created by the CPU 602 executing a program, andpasses them to CPU 602 by reading out the memorized data.

[0061] In the present embodiment, from CD-ROM 622 which has a programwhich realizes each functional unit of the image processing apparatus200 mentioned above, the program concerned is read and installed to thehard disk 618 in advance, and the functional unit of the above-mentionedimage processing apparatus 200 is realized by the CPU 602 reading theprogram concerned from the hard disk 618 and executing the program.

[0062] The program more specifically includes a distribution module forrealizing a distributing unit 220, a 2nd integrating module forrealizing the second integrating unit 240, a form calculation module forrealizing the shape computing unit 260, a 1st integrating module forrealizing the 1st integrating unit 280, a data processing module forrealizing the data processing unit 300, an image data generating modulefor realizing the image data generating unit 320, a gray densitygenerating module for realizing the gray-scale density data-generatingunit 340, and an outputting module for realizing the outputting unit360. Moreover, the program may be stored in the other recording media,such as the floppy disk (registered trademark) 620, MO and MD, otherthan the CD-ROM 622.

[0063] Next, referring to FIG. 4-FIG. 11, a 1st operation example of theimage processing apparatus 200 is explained. In this example, twohemispherical subjects 400 which stand in a line parallel and upward arecaptured with a projection-type image capturing apparatus, and3-dimensional image data is generated by the image processing apparatus200.

[0064] First of all, a step of acquiring gray-scale image is explained.As shown in FIG. 4, FIG. 5, and FIG. 6, a subject is captured from atleast three different angles (in this example, the direction of A, thedirection B, and the direction C), and projected image data, that is,gray-scale image 420, 440, and 460, are acquired. Here, it is desirableto make the angle interval of the direction A and the direction B, andthe angle interval of the direction A and the direction C into the equalvalue “theta” respectively, as shown in FIG. 4, FIG. 5 and FIG. 6.

[0065] Moreover, the direction B, and the direction C may be directed atthe angle rotated from the direction A with a same axis 401 as therotation center, or may be directed at the angle rotated respectivelyfrom the direction A with different axes 401 and 402 as rotationcenters.

[0066] Then, the gray-scale image 420, 440, and 460 are inputted intothe image processing apparatus 200. The gray-scale image 420, 440, and460 functions also as the 1st image data here.

[0067] In this example, the distributing unit 220 of the imageprocessing apparatus 200 generates gray-scale density-distributions 425,445, and 465, by spreading each of the gray-scale image 420, 440, and460 along the image-capturing direction, and the second integrating unitintegrates gray-scale density-distributions 425, 445, and 465, keepingeach angle, and generates 3-dimensional gray-scale data.

[0068]FIG. 7 shows an example of longitudinal-section information of the3-dimensional gray-scale data in the present embodiment. The3-dimensional gray-scaled at a show that object exists at the positionwhere all of the gray-scale density-distributions lap. FIG. 7 shows thata subject 400 exists where all of the gray-scale density-distributions425, 445, and 465 lap, that is, the part 470, 472, 474, 476, 478, and479. However, because the number of subject 400 originally is two, fourof the above six information are false information.

[0069]FIG. 8 shows another example of the longitudinal-sectioninformation on the 3-dimensional gray-scale data in the presentembodiment. In the present embodiment, densities of gray-scale image420, 440, and 460 differs, respectively. Moreover, gray-scaledensity-distributions 425, 445, and 465 are obtained by spreading thedensities of the gray-scale image 420, 440, and 460 within the capturedangles, and distributing the densities evenly. In this case, sincepositions other than data 470-479 may also have more than equivalentdensities to one of the data 470-479, the false information may increasefurther.

[0070] Moreover, the shape computing unit 260 of the image processingapparatus 200 calculates the shape data 265 which represent an outlineof the subject 400 from the gray-scale image 420, 440, and 460. Theshape computing unit 260 can not calculate whole outline of the subject400, but calculate the position commonly included in the range of visionof the direction A, the direction B, and the direction C in the figure.In this example, an outline of only a spherical part of the subject 400is calculated.

[0071] And the first integrating unit 280 of the image processingapparatus 200 superimposes the shape data by the shape computing unit260 on the 3-dimensional gray-scale image.

[0072]FIG. 9 shows longitudinal-section information on the integratedimage by the first integrating unit 280 in the present embodiment. Inthe present embodiment, two shape data 265 by the shape computing unit260 lap with data 470 and 472 respectively. Therefore, the dataprocessing unit 300 judges that the data 470 and 472 are true data thatexpress the object, and the data 474, 476, 478, and 479 are false data,then extracts only the data 470 and 472, and outputs them to thegray-scale density data-generating unit 340 and the outputting unit 360as 3-dimensional image data.

[0073] Consequently, the false data never be included in the3-dimensional image data.

[0074] Here, when the reliability of the shape data 265 is high, thedata processing unit 300 outputs the 3-dimensional image data whichconsidered the shape data 265. Moreover, when the gray-scaledensity-distributions 425, 445, and 465 are not in focus, the boundaryof the data 470 and the boundary of 472 do not become clear. In thiscase, the data processing unit 300 considers the data which exist withina distance defined in advance from the shape data 265, as the data 470and 472.

[0075] Moreover, human operation may substitute for the operation of thedata processing unit 300, and the false data may be removed by involvinga judgment of the human into the 3-dimensional image data by the dataprocessing unit 300, displaying a integrated data by the firstintegrating unit 280 on a display and the like. In this case, the falsedata can be removed corresponding to cases more properly.

[0076]FIG. 10 shows longitudinal section of the 3-dimensional image databy the data processing unit 300 in the present embodiment. It shows thatthe 3-dimensional image data by the data processing unit 300 arerecreating the subject 400 with sufficient accuracy.

[0077]FIG. 11 shows an example of operation of the gray-scale densitydata-generating unit 340. In this example, the gray-scale densitydata-generating unit 340 generates projected image, that is, gray-scaleimage 480, captured from just beside in the figure, that is, from adifferent angle from the gray-scale image 420, 440, and 460 inputtedwith the 3-dimensional data by the data processing unit 300. As shown inFIG. 11, the gray-scale density data-generating unit 340 can generatethe projected image captured from any specified angles, that is, thegray-scale image, and output it to the outputting unit 360.

[0078] Here, in the example of FIG. 11, the gray-scale image 480 whichthe gray-scale density data-generating unit 340 generated is not exactlysame as the gray-scale image which is obtained by seeing the subject 400from just beside. This is because the 3-dimensional image data by thedata processing unit 300 does not match the subject 400 exactly.

[0079] However, when the gray-scale image inputted into the imageprocessing apparatus 200 is the image by a transmission electronmicroscope, gray-scale image includes gradation even within an atom or amolecule. The 3-dimensional image data calculated become more closer tothe subject 400 by the second integrating unit 240 integrating thegray-scale image from further more angles, and the data processing unit300 deleting a position where the density is low in consideration of thegradation described above. Consequently, the gray-scale image which thegray-scale density data-generating unit 340 generates becomes stillcloser to the gray-scale image acquired by actual image.

[0080] And the outputting unit 360 outputs the shape data 265 by theshape computing unit 260, the 3-dimensional image data by the dataprocessing unit 300, that is, data 470, and data 472, and the gray-scaleimage by the gray-scale density data-generating unit 340 to a display ora printer as required.

[0081] Thus, in the first example of the operation, the image processingapparatus 200 acquires more exact 3-dimensional image data using thedistributing unit 220, the second integrating unit 240, the shapecomputing unit 260, the first integrating unit 280, and the dataprocessing unit 300.

[0082] Next, with reference to FIG. 12 and 13, the second example ofoperation of the image processing apparatus 200 is explained. In thisexample, the image processing apparatus 200 acquires 3-dimensional imagedata of the subject 400 using gray-scale image 420, 440, and 460 in thesame way as the first example of operation.

[0083] First, the shape computing unit 260 calculates shape data 265 asillustrated in FIG. 12. Then, the image data generating unit 320distributes the gray-scale image 420, 440, and 460 only over theperimeter of the shape data 265, as illustrated in FIG. 13, andconsiders parts where all of the three gray-scale image lap, that is,the data 470 and 472, as the 3-dimensional image data. Consequently, thefalse data never be included in the 3-dimensional image data.

[0084] Here, when the reliability of the shape data 265 is high, theimage data generating unit 320 outputs 3-dimensional image data whichconsidered the shape data 265. Moreover, when a boundary of the data470, and a boundary of the data 472 are not in focus, the image datagenerating unit 320 considers data that exist within a distance definedin advance from the shape data 265, as the data 470 and 472. Here, whenthe reliability of the shape data 265 is high, the image data generatingunit 320 outputs the 3-dimensional image data which considered the shapedata 265. Moreover, when the gray-scale density-distributions 425, 445,and 465 are not in focus, the boundary of the data 470 and the boundaryof 472 do not become clear. In this case, the image data generating unit320 considers data that exist within a distance defined in advance fromthe shape data 265, as the data 470 and 472. For example, when thicknessD of the subject 400 is known, densities of the gray-scale image 420,440, and 460 are distributed evenly within a range of the thickness Dfrom the outline. In this case, it is still more preferable for theabove-mentioned densities, to be distributed along the projectiondirection only within the range of thickness D.

[0085] Then, the gray-scale density data-generating unit 340 generates aprojected image captured from a different angle from the gray-scaleimage 420, 440, and 460 inputted with the 3-dimensional data by theimage data generating unit 320, that is, gray-scale image.

[0086] Thus, in the second example of operation, the image processingapparatus 200 acquires more exact 3-dimensional image data with theshape computing unit 260 and the image data generating unit 320.

[0087] As mentioned above, according to the image processing apparatus200, 3-dimensional image data that express a structural configuration ofa subject with more accuracy can be acquired by using the projectedimage data obtained by projecting the subject toward the plural angles.

[0088] The image processing apparatus 200 demonstrates theabove-mentioned merit especially when the range of angle, in which theimage processing apparatus 200 can capture an image, is limited. Forexample, when dealing with the image data acquired by a transmissionelectron microscope as the projected image data, the angle, in which themicroscope can capture an image, is limited. However, it becomespossible to clarify a structural configuration of a subject to amolecule level by using the image processing apparatus 200.

[0089] As a detailed example, there is a case where it is desirable toclarify a structure of a protein of cells and to clarify changesthereof. In this case, the 3-dimensional data which represent thestructural configuration of a protein, which is considered to be thesubject, can be acquired where false data, that is, so-called ghost areremoved, by capturing images of a pattern of the protein obtained byso-called quick-freezing deep-etch replica method by the transmissionelectron microscope, and processing the images with the image processingapparatus 200.

The Second embodiment

[0090]FIG. 14 shows a configuration of an image processing apparatus201, which is a second embodiment of the present invention. The imageprocessing apparatus 201 includes, as functional units, a sampling unit210, the distributing unit 220, the second integrating unit 240, theshape computing unit 260, the first integrating unit 280, the dataprocessing unit 300, the image data generating unit 320, the gray-scaledensity data-generating unit 340, and the outputting unit 360. Asampling unit 210 further includes an angle computing unit 215.

[0091] The image processing apparatus 201 is an apparatus which extractsgray-scale image of a plurality of subjects facing along a plurality ofangles respectively, included in a single gray-scale image obtained by aprojection-type image capturing apparatus such as an electron microscopeand the like, and acquires 3-dimensional data regarding a structuralconfiguration of the subject using a plurality of the extractedgray-scale image. Here, the subject is, as shown in FIG. 15 for example,a particle 491, which constitutes substance 490, which has a helicalsymmetry, such as a molecule which exists widely in a living world.

[0092] In the image processing apparatus 201, since operation offunctional units other than the sampling unit 210 and angle computingunit 2l5 is same as the operation of the image processing apparatus 200,which is the first embodiment, the explanation thereof is omitted.

[0093] When a single gray-scale image is inputted, the sampling unit 210extracts gray-scale image of a subject from the single gray-scale image,and outputs the image data to the distributing unit 220 and the imagedata generating unit 320. Here, when the above-mentioned singlegray-scale image includes gray-scale image of a plurality of subjects,the sampling unit 210 extracts the gray-scale image of a plurality ofthe subjects respectively. Moreover, when the 1st image data is thegray-scale image, the sampling unit 210 outputs the extracted gray-scaleimage also to the shape computing unit 260.

[0094] The angle computing unit 215 calculates, a direction of thesubject, that is, the angle, which the gray-scale image that thesampling unit 210 extracted indicates, and outputs the direction, withgray-scale image of the subject, to the distributing unit 220 and theimage data generating unit 320, and depending on cases, to the shapecomputing unit 260.

[0095]FIG. 16 shows an example of operation of the sampling unit 210 andthe angle computing unit 215. This example is the case where the singlegray-scale image is of the substance which has a helical symmetry, andbesides, the subject is a particle which constitutes the substance withthis helical symmetry.

[0096] First, the sampling unit 210 determines reference gray-scaledistribution data of a subject which is treated as a reference, byexternal input or by itself in the above described single gray-scaleimage (S10). Then, searches the same gray-scale image as the referencegray-scale distribution data from the single gray-scale image describedabove (S20). Here, the sampling unit 210 may consider gray-scale imagewithin a predetermined error the same gray-scale image.

[0097] And the sampling unit 210 extracts gray-scale images of aplurality of particles respectively which exist between the referencegray-scale distribution data and the gray-scale image searched in S20(S30), Here, as a method of recognizing the gray-scale image of aparticle, for example, there is a way of obtaining a distance of thegray-scale image and the reference gray-scale distribution data, anddividing the above described single gray-scale image for by each of thedistance.

[0098] Then, the angle computing unit 215 recognizes the number of thegray-scale images which exist between the gray-scale image searched inS20 and the reference gray-scale distribution data, that is, the numberof particles (S40), and calculates an angle by which the helix rotatesby one particle by dividing a rotation angle of the helix, for example,360 degrees, by the number of the particles recognized (S50).

[0099] Then, the angle computing unit 215 calculates angles which theparticles face by adding or subtracting the angle calculated by S50 tothe angle which the reference gray-scale distribution data represent(S60). Operation of S60 is repeated until the angles of all particlesare calculated (S70).

[0100]FIG. 17 shows another example of operation by the sampling unit210 and the angle computing unit 215. First, the sampling unit 210rotates a modeling image of the subject by predetermined angle (S100).And the sampling unit 210 extracts a part of the above mentioned singlegray-scale image (S110), and calculates the degree of image coincidenceas compared with the modeling image (S120). When it is judged that thedegree of the image coincidence is beyond a criteria (S130), a samplingunit 210 extracts the part concerned, recognizing it to be thegray-scale image of the subject (S140). The angle computing unit 215recognizes the angle of rotation by S100 as the angle which the subjectfaces (S150).

[0101] After repeating operations from S110 to S150 over the wholeregion of the above mentioned single gray-scale image (S160), thesampling unit 210 and the angle computing unit 215 repeat operationsfrom S100 to S160 until the angle reaches a predetermined angle (S170).

[0102] Thus, according to the image processing apparatus 201, since thesampling unit 210 and the angle computing unit 215 extract gray-scaleimage of a plurality of subjects facing different angles, from a singlegray-scale image, a plurality of gray-scale image are not necessarilyindispensable.

[0103] Therefore, when two or more images of substance, which has ahelical symmetry, are captured, according to the image processingapparatus 201, it becomes possible to acquire clearer 3-dimensionalimage data by using only the clearest gray-scale image. Moreover, evenwhen an image-capturing angle is limited due to physical restrictions ofsample creation or an experimental apparatus, it becomes possible toacquire 3-dimensional image data.

[0104] Although the present invention has been described by way ofexemplary embodiments, it should be understood that many changes andsubstitutions may be made by those skilled in the art without departingfrom the spirit and the scope of the present invention which is definedonly by the appended claims.

[0105] It is obvious from foregoing explanation that the presentinvention can provide an image processing apparatus, an image processingmethod, a computer readable medium, and a computer program thereof, foracquiring 3-dimensional image data that correctly express a structuralconfiguration of the subject by processing the projected image dataobtained by projecting the subject toward the plural angles.

What is claimed is:
 1. An image processing apparatus for acquiring3-dimensional data regarding a structural configuration of a subject byprocessing image data of the subject, comprising: a shape computing unitwhich calculates shape data which represent an outline of the subject,using a plurality of 1st image data of the subject obtained byprojecting the subject toward the plural angles; and a first integratingunit which integrates gray-scale image of the subject acquired by aprojection-type image-capturing apparatus, on said shape data.
 2. Animage processing apparatus as claimed in claim 1 further comprising: adistributing unit which generates gray-scale density-distributions byspreading and distributing densities shown in said gray-scale imagerespectively within angles along which said gray-scale image wascaptured; and a second integrating unit which generates 3-dimensionalgray-scale data, which express the subject with 3-dimensional imagedata, by integrating a plurality of said gray-scaledensity-distributions generated from a plurality of said gray-scaleimage captured from a plurality of angles, wherein said firstintegrating unit integrates said 3-dimensional gray-scale data by saidsecond integrating unit on said shape data.
 3. An image processingapparatus as claimed in claim 2, further comprising: a data processingunit which generates 3-dimensional image data by extracting only densitywhich exists in perimeter of a shape that said shape data represent,from said 3-dimensional gray-scale data.
 4. An image processingapparatus for acquiring 3-dimensional data regarding a structuralconfiguration of a subject by processing image data of the subject,comprising: a shape computing unit which calculates shape data whichrepresent an outline of the subject, using a plurality of 1st image dataof the subject obtained by projecting the subject toward the pluralangles; and an image data generating unit which generates 3-dimensionalimage data by distributing density of the gray-scale image of thesubject acquired by a projection-type image-capturing apparatus, aroundsaid outline.
 5. An image processing apparatus as claimed in claim 3 or4, further comprising: a gray-scale density data-generating unit whichcalculates from said 3-dimensional image data, said gray-scale imagewhich should be acquired when the subject was captured from a differentangle from said plurality of the angles.
 6. An image processingapparatus as claimed in claim 1 or 4, wherein said gray-scale image isimage data acquired by a transmission electron microscope.
 7. An imageprocessing apparatus as claimed in claim 1 or 4, wherein image data ofthe pattern of said subject is used as said first image data.
 8. Animage processing apparatus as claimed in claim 1, further comprising: asampling unit which extracts respectively gray-scale image of aplurality of said subjects facing a plurality of angles, included in asingle gray-scale image acquired by the projection-type image-capturingapparatus, wherein said first integrating unit uses a plurality of saidgray-scale image which said sampling unit extracted.
 9. An imageprocessing apparatus as claimed in claim 4, further comprising: asampling unit which extracts respectively gray-scale image of aplurality of said subjects facing a plurality of angles, included in asingle gray-scale image acquired by the projection-type image-capturingapparatus, wherein said image data generating unit uses a plurality ofsaid gray-scale image which said sampling unit extracted.
 10. An imageprocessing apparatus as claimed in claim 8 or 9, wherein said subjectconstitutes substance which has a helical symmetry, said apparatusdealing with gray-scale image of said substance as said singlegray-scale image, further comprising: an angle computing unit whichacquires a reference angle, which is an angle the single subject as areference faces, and further calculates angles which each of saidsubjects faces by adding or subtracting the angle which the helixrotates by said single subject to said reference angle.
 11. An imageprocessing apparatus as claimed in claim 1 or 4, wherein said shapecomputing unit includes: a position relating unit which relatespositions of processing objective points in said subject in a 1st and2nd images which are consecutive in the order of angle, and relates thepositions of the processing objective points in said 2nd image and a 3rdimage; a height data computing unit which calculates height data forrepresenting height of the position of said processing objective pointusing 2 sets of position relation by said position relating unit; and ashape data computing unit which generates said shape data of saidsubject using said height data of a plurality of processing objectivepoints calculated by said height data computing unit, wherein saidapparatus calculates said shape data using three or more said firstimages.
 12. An image processing apparatus as claimed in claim 11,wherein said 1st, 2nd, and 3rd images ware images captured with arotation of said subject by predetermined angle with a same rotationaxis as the rotation center, and said height data computing unitcalculates said height data using said predetermined angle.
 13. An imageprocessing apparatus as claimed in claim 11, wherein said 1st and 2ndimage are images captured with a rotation of said subject bypredetermined angle with a first rotation axis as the rotation center,and said 3rd image is an image captured with a rotation of said subjectby said predetermined angle from the position where said 1st image wascaptured with another rotation axis as the rotation center, and saidheight data computing unit calculates said height data using saidpredetermined angle.
 14. An image processing apparatus as claimed inclaim 11, wherein said position relating unit calculates distances insaid image from a base line defined beforehand in said subject to saidprocessing objective points, and re-detects positions of said processingobjective points in said 1st and 2nd image, when the difference of saiddistance in said 1st image and said distance in said 2nd image is morethan a predetermined value, and relates said processing objective pointsin said 2nd and 3rd image again when the difference of said distance insaid 2nd image and said distance in said 3rd image is more than apredetermined value.
 15. An image processing apparatus as claimed inclaim 11, wherein said height data computing unit calculates said heightdata by a least-squares error method with height data of said 1st and2nd image and height data by said 2nd and 3rd image.
 16. An imageprocessing apparatus as claimed in claim 15, wherein said positionrelating unit re-detects the positions of said processing objectivepoints, when a error by said least-squares error method is more than apredetermined value.
 17. An image processing method for acquiring3-dimensional data of a subject by processing image data of the subject,comprising steps of: calculating shape data which represent an outlineof the subject, using a plurality of 1st image data of the subjectobtained by projecting the subject toward the plural angles; generatinggray-scale density-distributions by spreading and distributing densitiesshown in gray-scale image of the subject acquired by a projection-typeimage-capturing apparatus, respectively within angles along which saidgray-scale image was captured; generating 3-dimensional gray-scale data,which expresses the subject with 3-dimensional image data, byintegrating a plurality of said gray-scale density-distributionsgenerated from a plurality of said gray-scale image captured from aplurality of angles; integrating said 3-dimensional gray-scale data onsaid shape data; and generating 3-dimensional image data by extractingonly density which exists in perimeter of an outline that said shapedata represent, from said 3-dimensional gray-scale data.
 18. An imageprocessing method for acquiring 3-dimensional data of a subject byprocessing image data of the subject, comprising steps of: calculatingan outline of the subject, using a plurality of 1st image data of thesubject obtained by projecting the subject toward the plural angles; andgenerating 3-dimensional image data by distributing respectively densityof the gray-scale image of the subject acquired by a projection-typeimage-capturing apparatus, around said outline.
 19. An image processingmethod as claimed in claim 17, further comprising steps of: extractinggray-scale image of a plurality of said subjects facing a plurality ofangles from a single gray-scale image acquired by the projection-typeimage-capturing apparatus, and generating said gray-scaledensity-distributions using said extracted gray-scale image.
 20. Animage processing method as claimed in claim 18, further comprising stepsof: extracting gray-scale image of a plurality of said subjects facing aplurality of angles from a single gray-scale image acquired by theprojection-type image-capturing apparatus, and generating 3-dimensionalimage data by distributing respectively said extracted gray-scale image,around said outline.
 21. An image processing method as claimed in claim19 or 20, wherein said subject constitutes substance which has a helicalsymmetry, while said apparatus deals with gray-scale image of saidsubstance as said single gray-scale image.
 22. An image processingmethod as claimed in claim 21, further comprising steps of: recognizinga reference angle, which is an angle the single subject as a referencefaces, and calculating angles which each of said subjects faces byadding or subtracting the angle which the helix rotates by said singlesubject to said reference angle.
 23. A computer-readable medium whichstores therein a program for image processing for acquiring3-dimensional data of a subject by processing image data of the subject,said program comprising: a shape computing unit operable to calculateshape data which represent an structural configuration of the subject,using a plurality of 1st image data of the subject obtained byprojecting the subject toward the plural angles; and a first integratingunit operable to integrate gray-scale image of the subject acquired by aprojection-type image-capturing apparatus, on said shape data.
 24. Acomputer-readable medium which stores therein a program for imageprocessing for acquiring 3-dimensional data of a subject by processingimage data of the subject, said program comprising: a shape computingunit operable to calculate an outline of the subject, using a pluralityof 1st image data of the subject obtained by projecting the subjecttoward the plural angles; and an image data generating unit operable togenerate 3-dimensional image data by distributing density of thegray-scale image of the subject acquired by a projection-typeimage-capturing apparatus, around said outline.
 25. A computerexecutable program for image processing for acquiring 3-dimensional dataof a subject by processing image data of the subject, said programcomprising: a shape computing unit operable to calculate shape datawhich represent a structural configuration of the subject, using aplurality of 1st image data of the subject obtained by projecting thesubject toward the plural angles; and a first integrating unit operableto integrate gray-scale image of the subject acquired by aprojection-type image-capturing apparatus, on said shape data.
 26. Acomputer executable program for image processing for acquiring3-dimensional data of a subject by processing image data of the subject,said program comprising: a shape computing unit operable to calculate anoutline of the subject, using a plurality of 1st image data of thesubject obtained by projecting the subject toward the plural angles; andan image data generating unit operable to generate 3-dimensional imagedata by distributing density of the gray-scale image of the subjectacquired by a projection-type image-capturing apparatus, around saidoutline.